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Abstract 

This paper reports on a pilot study that examined the use of a science and technology 
curriculum based on robotics to increase the achievement scores of youth ages 9-1 1 in an after 
school program. The study examined and compared the pretest and posttest scores of youth in 
the robotics intervention with youth in a control group. The results revealed that youth in the 
robotics intervention had a significant increase in mean scores on the posttest and that the 
control group had no significant change in scores from the pretest to the posttest. In addition, 
the results of the study indicated that the evaluation instrument used to measure achievement 
was valid and reliable for this study. (Keywords: robotics, 4-H, informal learning, science 
achievement, experiential learning. ) 

INTRODUCTION 

Robots have left assembly lines and research labs and arrived on the doorstep of 
education. Some educators have claimed that through hands-on experimentation, 
robots help youth transform abstract science, engineering and technology (SET) 
concepts into concrete real-world understanding. Recent improvements in cost 
and simplicity make it possible for students to engage in this kind of hands-on 
experimentation with robots. Nebraska 4-H has begun investigating this potential 
robotics holds for improving SET education. Nebraska 4-H implemented a new 
robotics curriculum in an after school program and evaluated it using a new testing 
instrument based on the stated learning objectives in the curriculum. 

REVIEW OF RELATED LITERATURE 
Robotics in the Classroom 

The United States’ economy is highly dependent on advanced technology. 
Technology and related innovation are responsible for at least half of U.S. economic 
growth (Bonvillian, 2002). Industries that rely on technology need new scientists 
and engineers every year to help propel their success and it is up to those in our 
schools to produce these graduates. Unfortunately, U.S. students are less prepared 
than many other first-world countries in terms of science and math. At the fourth 
grade level, U.S. students are competitive in science but fall behind most first-world 
countries in math (Gonzales, Guzman, Partelow, Pahlke, Jocelyn, Kastberg, & 
Williams, 2004). By age fifteen, U.S. students are still relatively poor math per- 
formers and fall behind the international average in science literacy as well (Lemke, 
Sen, Pahlke, Partelow, Miller, Williams, Kastberg, & Jocelyn, 2004). If innovation is 
going to continue to drive the United States’ economy, its educational system must 
improve these scores and entice graduates into SET careers (Bonvillian, 2002). 
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One new approach to improving SET education that is gaining popularity 
is the use of robots to teach the content. Advances in technology have brought 
down the cost of robots and made it easier to bring them into classrooms with 
tight budgets. Seymour Papert (1980) laid much of the groundwork for using 
robots in the classroom in the 1970s. Breaking with traditional computer aided 
instruction models where computers essentially programmed children, Papert 
attempted to create an environment where children programmed computers and 
robots. In doing so, the children could gain a sense of power over technology. He 
believed that children could identify with the robots because they are concrete, 
physical manifestations of the computer and the computers programs. Other 
researchers have also identified the concrete nature of robots as being one of their 
important advantages. By testing scientific and mechanical principles with the 
robots, students can understand abstract concepts and gain a more functional level 
of understanding (Nourbakhsh, Crowley, Bhave, Hamner, Hsium, Perez-Bergquist, 
Richards, & Wilkinson, 2005). Students can also learn that in the real world there 
is not necessarily only one correct answer to every question (Beer, Chiel, & Drushel, 
1999). Beer et al. (1999) felt that it was more important for their students to come 
up with creative solutions to problems than it was to recite answers they learned in 
class by rote. 

Another argument for teaching children with robots is that they see the robots as 
toys (Mauch, 2001). In fact, one widely used kit of robotic equipment is made by 
Lego, a well-known manufacturer of childrens building block toys. Children using 
this kit can build and program robots out of the same materials they have in their 
toy chests at home. This makes anything they learn with the kits seem entertaining 
as well. 

Research also suggests that robots tie into a variety of disciplines. A robot is made 
of component parts of motors, sensors and programs. Each of these parts depends 
on different fields of knowledge such as engineering, electronics, and computer 
science. This interdisciplinary nature of robots means that when students learn 
to engineer robots they will inevitably learn about the many other disciplines 
that robotics utilize (Papert, 1980; Rogers & Portsmore, 2004). In the same 
way, teaching students how to build robots teaches them how all the parts of a 
complex system interact and depend upon each other (Beer et al., 1999). This is 
an important lesson for computer scientists, biologists, doctors or anyone who will 
ultimately need to understand complex systems. 

Early adopters of robotics in the classroom have reported many accolades; 
however, there is a clear lack of quantitative research on how robotics can increase 
STEM achievement in students. Most research involving robotics in the classroom 
was conducted with high school and college age students with results dependent 
on teacher or student perceptions rather than rigorous research designs based on 
student achievement data. 

The case studies which exist in the literature positively document the use of 
robotics to teach a variety of subjects to a wide array of age groups. They illustrate 
the potential effectiveness of robotics to positively impact both learning and 
motivation (Fagin & Merkle, 2003). Studies show that robotics generates a high 
degree of student interest and engagement and promotes interest in math and 

230 Spring 2007 : Volume 39 Number 3 

Copyright © 2007, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 


science careers (Barnes, 1999; Robinson, 2005; Rogers & Portsmore 2004). The 
robotics platform also promotes learning of scientific and mathematic principles 
through experimentation (Rogers & Portsmore, 2004), encourages problem 
solving (Barnes, 1999, Mauch, 2001; Nourbakhsh et al., 2005; Robinson, 2005; 
Rogers & Portsmore, 2004) and promotes cooperative learning (Beer et al., 1999; 
Nourbakhsh et al., 2005). Despite the positive instructional and motivational 
benefits these studies suggest, rigorous quantitative research is missing from the 
literature. 

In the classroom, some educators have used robots as a tool to assist in the 
teaching of actual programming languages (Barnes, 2002; Fagin & Merkle, 2003). 
For example, Fagin and Merkle (2003) and Barnes (2002) used robots to help teach 
the programming languages of ADA and Java, respectively. The main emphasis for 
their courses was on teaching the programming languages and basic programming 
structures over the engineering and mechanical aspects of robots. Other courses 
that use robots have focused on the construction and programming of the robots 
themselves (Beer et al., 1999; Nourbakhsh et al., 2005). 

Moore (1999) used robots to teach her fourth-grade students several different 
topics under the umbrella of examining robots. She used the topic as a “hook” to 
capture her students’ attention; then she weaved other disciplines into this central 
theme and asked her students to think critically about robots. According to Moore 
(1999) students will build and program robots, understand geometry concepts, 
write and share stories with peers and compare and contrast technology systems 
with human body systems. The study does not provide a quantitative evaluation 
of the robotics program. Rogers and Portsmore (2004) also taught young students 
using robots. They designed a curriculum using LEGO robots that teaches 
kindergarten though fifth-grade students about engineering. 

Most of the literature describing the use of robots to teach science and technology 
reports positive impacts on what their students learned about SET. Several 
researchers reported that learning with robots is more interesting and improves 
students’ attitudes about SET subjects (Fagin & Merkle, 2003; Mauch, 2001; 
Robinson, 2005). Some researchers noted that female students in particular 
are more likely to appreciate learning with robots than traditional SET teaching 
techniques (Nourbakhsh et al., 2005; Rogers & Portsmore, 2004). 

Learning with robots helps teach scientific and mathematic principles through 
experimentation with the robots. Rogers and Portsmore (2004) reported success 
in teaching decimals at the second grade level by making a robot move for a time 
between one and two seconds. Papert (1980) used robots to teach geometry 
concepts. Robots helped his students see the relationships between programming, 
mathematics, and movement of the robot. Building and programming robots also 
requires that the students develop problem solving skills (Beer et al., Mauch, 2001; 
Nourbakhsh et al., 2005; 1999). Beer et al. (1999) emphasized that designing 
an entire system that was needed to work in the real world required problem 
solving skills that would serve them well in their future careers no matter what 
discipline they chose. Teamwork is another career skill that robots appear to foster. 
Nourbakhsh et al. (2005) and Beer et al. (1999) identified teamwork as being 
important outcomes of their robotics courses. 
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Not all the reported results of using robotics are positive, however. In one of the 
very few quantitative studies that examined the use of robots’ effectiveness in the 
classroom with test scores, Fagin and Merkle (2003) found that robots did not help 
introductory computer science students learn to program. The students taught 
using robots did significantly worse than students not taught with robots. Fagin 
and Merkle reported that there were limitations to the way they were implemented 
that might have counteracted the helpfulness of the robots, but it is clear that while 
robots have positive potential, they are no panacea. 

THE 4-H ROBOTICS PROGRAM: AN OVERVIEW 

In the winter of 2005 Nebraska State 4-H teamed with an elementary school 
in a small rural town in central Nebraska to pilot test educational robotics in 
an after school program. The intervention used in this program is composed 
of a newly developed National 4-H Cooperative Curriculum System (CCS) 
robotics curriculum and a kit of robotic components from LEGO, called LEGO 
Mindstorms. The LEGO Mindstorms kit is comprised of 828 parts including axles, 
gears, motors and sensors. The kit includes a programmable microcomputer with 
three output and three input ports for controlling sensors and motors. In addition, 
the robots are programmed using a specialized programming language called 
ROBOLAB. The 4-H robotics curriculum contains 28 lessons designed around the 
Mindstorms kit. It begins with simple building and programming challenges and 
culminates in advanced robotic programming and engineering topics. 

Nebraska 4-H began the robotics intervention by training some of the after 
school personnel with the robotics and curriculum. The training focused on the 
fundamentals of the program. When the training was complete, the participants 
knew the primary components of the robotics kit, had built their first robot, and 
had learned how to write simple programs for the robot with the ROBOLAB 
programming environment. After the initial training, the adult leaders were allowed 
to take home the robotics kit and curriculum and work through the rest of the 
materials on their own over the course of a few weeks. When the after school 
program began, the teacher and two or three adult assistants were able to lead small 
groups of children through the activities. 

THEORETICAL FRAMEWORK 

4-H s robotics curriculum was designed around the experiential learning model, 
which is built on Kolbs (1984) experiential learning theory. The model has five 
phases: 1) experience — do the activity, 2) share — reactions and observations 
in a social context, 3) process — analyze and reflect upon what happened, 4) 
generalize — discover what was learned and connect to life, and 5) apply — what was 
learned to a similar or different situation (Woffinden & Packham, 2001). 

Experiential learning distinguishes 4-H youth development education from 
many formal education methods. Youth are first provided an opportunity to learn 
before being told or shown how and then share what they did, consider what was 
important about what they did, generalize the experience to their own lives and 
finally apply what they learned to a new situation. Each activity of the curriculum 
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begins with a brief overview of the topics covered in the activity followed by a 
challenge for the youth to complete. After the youth have completed the challenge 
they are prompted to answer questions that cause them to reflect and generalize on 
their experiences. 

Experiential learning is based on a constructivist theory that purports that 
learning is an active process in which much of what an individual learns and 
understands is constructed by integrating new knowledge with existing knowledge. 
It is similar in principle to problem-based learning in that students learn concepts 
and principles through authentic experiences and problems; learning occurs in small 
groups; and teachers act as facilitators (Barrows, 1996). While procedural knowledge 
is provided, students are encouraged to transfer such knowledge to similar and 
different situations. Students who learn in this manner are responsible for their own 
learning, seek out new knowledge and are better prepared to generalize knowledge 
(Pressley, Hogan, Wharton-McDonald, Misretta, & Ettenberger, 1996). This 
approach results in better long-term content retention than traditional instruction 
(Norman & Schmidt, 1992), higher motivation (Albanese & Mitchell, 1993), and 
the development of problem-solving skills (Hmelo, Gotterer, & Bransford, 1997). 
Research also indicates that experiential education enhances social and academic 
development among children by encouraging social interaction and cooperative 
learning (Deen, Bailey, & Parker, 2001; Slavin, 2000). 

Papert (1980) found that robots were an excellent way to put constructivist 
theory into practice. The children learning with robots were able to imagine 
themselves in the place of the robot and understand how a computers 
programming worked. The children were able to transfer their understanding of 
the real world into comprehension of logic and mathematical principles. Papert 
believed that what makes many concepts difficult for children to understand is 
a lack of real-world materials that demonstrate the concept. He believed that 
programmable robots were flexible and powerful enough to be able to demonstrate 
ideas that previously had no easy real-world analogy. 

PURPOSE AND RESEARCH QUESTIONS 

The theoretical framework that guides this research is founded in the 4-H 
experiential learning model based on Kolbs (1984) theory of experiential learning. 
This study looked for quantitative data that describes how an experiential, robotics- 
based curriculum affected youths’ understanding of SET topics. 

The main purpose of this study was to determine the effects of an informal 4-H 
experiential science intervention based on robotics in an after school environment 
on levels of achievement in science, engineering, and technology (SET) for youth 
ages nine to eleven. In addition, the research intended to validate a 24-item 
multiple-choice assessment instrument to measure general SET domain knowledge 
and specific domain knowledge based on the stated learning objectives in the 
curriculum. This study compares the achievement of youth who participated in 
the robotic intervention with those that did not participate in the intervention. 
Specifically, the following research questions were addressed: 

1 . What is the validity and reliability of the assessment instrument 
developed for this study? 
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2. What is the impact of the robotics instruction in promoting student 
learning in science, engineering, and technology (SET) for youth ages 
nine to eleven at an after school program? 

METHOD 

To determine the effectiveness of the intervention a pretest/posttest quasi- 
experimental study with a control group was designed. The control group did not 
participate in the after school robotics program and did not have access to the 
robotic kits or computers. The experimental group participated in the robotics 
program twice a week for one hour over six weeks. The pretest was administered to 
each group prior to the beginning of the intervention. After six weeks the posttest 
was administered to both groups. The evaluation instrument was a paper-and-pencil 
based, 24-item assessment instrument with one right answer and three distracters 
per question created by the researchers. Each assessment question was derived from 
activities within the 4-H robotics curriculum. 

Participants 

The participants for the study were all from the same Nebraska rural elementary 
school. The overall sample (including both the experimental and a comparison 
groups) contained 32 students, with an age range of 9-1 1 years (median age 
was 9.00). A group of 14 students (65% male, 35% female) represented the 
experimental group and were selected to participate in the robotics intervention 
based on their participation in the after school program. To help provide some 
similarity in a comparison group, 18 additional students (63% male, 38% female) 
were randomly selected by the lead educator from the remaining students in the 
school, who were not participating in the robotics intervention. 

Procedure 

A national writing team comprised of youth development specialists from 4- 
H and content experts from Carnegie Mellon University’s Robotics Academy 
developed the 4-H Robotics curriculum. The curriculum features two activity 
manuals with 12 lessons per manual and a leaders’ guide. For the intervention, 
the after school youth were broken into groups of 4 to 5; each group had an adult 
volunteer or after school teacher to lead the activities. 

Students were introduced to robotics by building a basic “tankbot,” a LEGO 
robot that has two motors attached to two tank-like treads. Next, the youth learned 
to program the “tankbot” using the ROBOLAB software and advanced through 
increasingly complex programming tasks. For example, once students learn to 
turn their robots, they are introduced to the concept of “calibration,” where they 
determine how long it takes the robot to turn 90 degrees. Another activity had 
the students program their robots to navigate a maze with several left and right 
90-degree turns. Students learned to loop sections of their programming code by 
having their robot race around a 36" by 36" square racetrack three times. Finally, 
students fitted their robots with touch and light sensors and programmed the robots 
to react to changes registered by the sensors. 
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Instrument Validation 

Prior to administration of the evaluation instrument, two experts from Carnegie 
Mellon University’s Robotics Academy reviewed the relevance and validity of 
each item. The expert reviewers were selected based on their knowledge of robotic 
engineering and their success in developing curriculum based on robotics to 
teach SET topics. Together, these experts have published more than 12 curricular 
programs and led the effort to develop the 4-H robotics curriculum used in this 
intervention. Modifications to the instrument were done based on the expert 
reviews. 

To determine the internal consistency estimates of reliability a Cronbachs alpha 
reliability coefficient was calculated for the posttest. The alpha coefficient for the 
posttest was 0.86 on 24 items. Because the assessment instrument covered topics 
specific to the ROBOLAB program that youth in the control group would not have 
been exposed to, the ROBOLAB specific questions were separated. A Cronbachs 
alpha was calculated for SET concepts (alpha = .764) and the ROBOLAB concepts 
(alpha = .750). A Kuder-Richardson (KR20) reliability measure was calculated to 
determine the stability of the scores on the posttest. Said another way, the KR20 
score indicates if the youth would score the same if they took the test again. An 
acceptable KR20 measure is 0.7 or greater. The KR20 score for the posttest was 
0.87, indicating an 87% probability of achieving the same score if the student took 
the test again. 

The information in Table 1 displays an overview of the perceived value of each 
posttest question based on P-values, point biserial correlations, and an analysis of 
distractors chosen by the students. Experimental and comparison group scores are 
combined. Questions 12, 13, 19, and 21 had low P-values (.00 to .16) indicating 
the questions were extremely difficult. Moreover, questions 19, 21 and 22 had 
negative point biseral values indicating that higher performing students missed the 
questions more often than lower performing students. In addition, a proportion 
value (P-value) was established for each question on the posttest. The P-value 
represents the proportion of students answering the question correcdy. Items that 
are shaded in dark gray have a low P-value under 0.25 and may be questions that 
are extremely difficult (see Table 1, page 236). 

Moreover, a Point Biserial value was calculated for each item to determine item 
discrimination between high-scoring and low-scoring examinees. The Point Biserial 
calculation is used to determine the relative quality of the assessment questions. The 
Point Biserial score should range from 0.3 to 0.7. A negative Point Biserial score 
indicates that high-scoring examinees missed that question (see Table 1). Point 
Biserial values from 0.3 are shaded in light gray and negative Point Biserial scores 
have a black background. These values indicate questions where higher performing 
students scored lower than lower performing students. Finally, the table displays 
the distractors for each question and the percentage of students that selected the 
distractor; correct answers are in bold (see Table 1). 

RESULTS 

The data were analyzed using SPSS for Windows v. 13. Two sets of data from 
the experimental group were excluded because the posttest was incomplete. An 
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Table 1: Posttest — Proportion of Correct Answers (p-values) and Point Biserial Scores by Question Number Groups Combined 
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independent t-test was conducted on each exam question to compare the means 
scores between the control and experimental group (see Table 2, pp. 238-239). 

The posttest questions are broken down by SET concepts and the ROBOLAB 
computer programming interface. 

To examine the group performance on each question, Pearsons correlation 
coefficient was calculated to indicate the relative effect size. A Pearsons correlation 
coefficient as a measure of effect can lie between 0 (no effect) to 1 (a perfect effect) 
(Field, 2001). In addition, the difference of mean scores between groups on the 
posttest was calculated as was the percent change score (see Table 3, page 240). 

To determine the effectiveness of the intervention overall, pretest and posttest 
scores were compared between the control and the experimental group using an 
independent t-test. ALevenes test for the homogeneity of variance F(30) = 1 . 52, 

= .227 on the pretest indicated that the variances between mean scores were equal. 
However, a significant difference was detected on the posttest mean scores F(30) = 
10.8 A,p < .003 indicating that the variance of the posttest means were not equal 
between the control and experimental group and therefore equal variances were 
not assumed. The results of the pretest mean scores between the control group 
(M= 7.50, SD = 2.58) and the experimental group (M= 7.93, SD= 3.71) were 
not significant r(30) = 11.60, p = .702. To determine if there was a significant 
difference between the posttest scores for the control group and experimental 
group an additional independent-samples t-test was conducted. The results of the 
posttest mean score between the control group {M = 7.44, SD = 2.98) and the 
experimental group (M = 17.00, SD = .88) was significant r(22.17) = 12.93, p < 
.000. A Pearsons correlation coefficient r was calculated to determine the effect 
size of the intervention. The results indicate a large effect r = .943 where *(20.67) = 
12.93. Boxplots showing the pretest and posttest mean scores by group are shown 
in Figure 1 (page 241). 

DISCUSSION 

The purpose of this study was to investigate the effectiveness of an informal 4- 
H science curriculum to teach SET concepts. The results of the study based on 
the increase of mean scores from the pretest to the posttest for the experimental 
group indicate the robotics was effective at teaching youth about SET concepts 
like computer programming, robotics, mathematics, and engineering. The overall 
effect size for the intervention was calculated at .943, which indicates a large effect 
from the robotics program. The overall percent change from the control group [M 
= 7.44, SD - 2.98) to the 4-H robotics group (M= 17.00, SD = .88) was 128%. 
Moreover, there was no significant difference between the control groups pretest 
and posttest scores, while the robotics group had a significant increase from the 
pretest (M = 7.93, SD= 3.71) to the posttest (M= 17.00, SD = .88) t( 14) = 8.95, p 
< .000. The mean difference was 9.07 between the pretest and the posttest for the 
robotics group. 

The second purpose of the study was to validate an assessment instrument to 
document the degree to which students can recognize SET concepts taught in the 
4-H Robotics curriculum. The results of the Cronbach’s alpha (.86) indicate that 
the instrument is reliable. In addition, the KR20 score .87 indicates a similar high 
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Table 2: Mean Scores and Standard Deviations by Question by Group for the Pretest and Posttest with T-Scores and Significance 
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Table 3: Posttest Questions with T-Scores, Effect Size, Mean Difference, and 
Percent Change in Mean Scores 





M 


Posttest Question 

t 

Size 1 

Diff 

% A 

A robot must be in order to move 

2.204 

.47 

.22 

28 

A programming loop does which of the following 

4.537 

.64 

.64 

291 

The following is a(n) in ROBOLAB 

(Image of Icon) ” 

5.831 

.82 

.67 

203 

What enables a robot to interact with its environment? 

5.831 

.82 

.67 

203 

What icons are needed in every ROBOLAB program? 

3.289 

.62 

.64 

64 

What is a computer program? 

6.648 

.85 

.72 

257 

How does the RCX communicate with your computer? 

7.714 

.88 

.78 

355 

If you did not know what an icon did within ROBOLAB 
how would you find out? 

2.522 

.45 

.42 

191 

What does a robot have that a machine does not? 

3.688 

.67 

.44 

79 

What is a ratio? 

0.462 

.09 

.08 

29 

If a plate is 1/3 as think as a brick how many plates would 
you need to equal one brick? 

6.648 

.85 

.72 

257 

What does firmware on the RCX do? 


0 

0 

0 

Collecting information about how far your robot 
will travel in a given amount of time and using the 
information to estimate how long it will take the robot to 
go a eiven distance is called 

-0.18 

.03 

-.03 

-18 

What is pseudocode? 

3.907 

.60 

.54 

138 

When programming vour robot a fork is used to 

5.831 

.82 

.67 

20. 

What does the math symbol < mean? 

3.493 

.54 

.53 

161 

If you had a light sensor reading of 30 for dark and 50 for 
light what should the threshold value be? 

11.662 

.94 

.89 

809 

Which would be an example of multi-tasking? 

3.907 

.60 

.54 

138 

The rotation sensor works like what on a car? 

-1.844 

-.41 

-.17 

-100 

What is the primary purpose of gears? 

4.415 

.64 

.60 

182 

Which gear ratio will permit your robot to cover 3 feet 
the fastest? 

-0.827 

-.15 

-.10 

-59 

Which gear ratio has the most torque? 

-0.913 

-.17 

-.15 

-34 

In computer programming what is a variable or container 
icon used for? 

2.05 

.38 

.35 

159 

A programming subroutine is used when 

0.861 

.17 

.15 

54 


Note: 1 The effect size of change can be interpreted by the follow numbers, 
r = .10 (small effect) 
r = .30 ( medium effect) 
r = .50 (large effect) 
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Figure 1. Boxplots of student scores on the pretest and posttest by group 


reliability. Two robotics experts were called on to validate the questions used in the 
instrument. 

While there was a vast improvement on the posttest for the robotics group there 
were a few questions where the control group outscored the robotics group. Three 
questions, 19, 21, and 22 had negative Point Biserial values indicating the lower 
performing students outperformed higher-level students on these questions. One 
explanation for the lower scores on those particular items is related to the limitation 
that the robotics group did not get completely through the full curriculum, and 
thus those particular concept questions were not supported with instruction. 

This explanation is plausible since the lead educator mentioned that the Robotics 
group, due to time restraints, did not complete the entire curriculum as originally 
planned. In addition, it may be that the control groups more generalized classroom 
instruction did have at least some partial contribution to their scores on those same 
items. For example, a portion of the comparison group might have had limited 
classroom instruction on ratios and gears explaining their higher scores on questions 
19,21 and 22. 

While the assessment instrument was deemed valid and reliable it may not be 
useful outside of the scope of this study due to the specificity of the questions. Some 
questions were general in scope and could be used in other assessment instruments. 
However, many questions were very specific and tied to the stated learning 
objectives of the 4-H robotics intervention and thereby limited the usefulness of the 
assessment instrument. Conversely, the assessment instrument did provide a means 
to quantitatively measure the achievement of students. 

Journal of Research on Technology in Education 24 1 

Copyright © 2007, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 




CONCLUSION 

Overall, the findings of this study support the use of 4-H robotics to teach SET 
concepts in an after school program and that the evaluation instrument developed 
to test the SET concepts is reliable and valid. More research is needed to determine 
the effectiveness of the robotics program with different populations. In addition, 
research is needed to determine if the mean scores will increase on questions 19, 

21, and 22 when the youth complete the entire curriculum. Another question 
to look at is the effectiveness of using robotics in an informal environment like 
traditional 4-H clubs led by adult volunteers and extension personnel in non-school 
environments (at home or extension office) that meet on evenings and weekends. 
An additional research area is the effectiveness of the 4-H Robotics program with 
individual youth working with a parent in the home. Moreover, research is needed 
to examine whether the program helps fosters positive attitudes towards SET in 
school and as a career. 
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